Harvesting comparable corpora and mining them for equivalent bilingual sentences using statistical classification and analogy- based heuristics

摘要

Parallel sentences are a relatively scarce but extremely useful resource formany applications including cross-lingual retrieval and statistical machinetranslation. This research explores our new methodologies for mining such datafrom previously obtained comparable corpora. The task is highly practical sincenon-parallel multilingual data exist in far greater quantities than parallelcorpora, but parallel sentences are a much more useful resource. Here wepropose a web crawling method for building subject-aligned comparable corporafrom e.g. Wikipedia dumps and Euronews web page. The improvements in machinetranslation are shown on Polish-English language pair for various text domains.We also tested another method of building parallel corpora based on comparablecorpora data. It lets automatically broad existing corpus of sentences fromsubject of corpora based on analogies between them.